Haptic communications

ABSTRACT

A haptic output device in combination with a network has its effective inertia increased by a controlling computer algorithm whereby potential latency effects of the network are damped out.

The present invention relates to haptic communications and moreparticularly to improving the response of haptic devices coupled by wayof a telecommunications network.

Tactile output from computers has been used to enhance game playing toprovide a “feel”, for example vibration, thus adding an additionalsensory perception. to the games. Such outputs have also been used toenable visually impaired people to read documents and to feel drawingsand the like. The basic operation of haptics output devices has beendescribed in our co-pending European Patent Application No. 01305947.2which disclosed a method for adapting haptic interface outputcharacteristics to correct for per-person differences in the sense oftouch. In a further European patent application No. 01307951.2 there isdisclosed a method of enabling reading of the Moon alphabet by use of ahaptics output device. In the transmission of character sets fromcomputers or data stores to haptics output devices there is unlikely tobe any time critical activity dependant upon the output signals.However, where game play is involved, particularly if players arecompeting against each other or against the machine in a competitivemanner, transmission delays of forward or reverse force parameters mayhave a significant impact on the sensed experience.

As game play is more likely to be carried out over a connectionlessnetwork, for example the internet or world wide web, rather than by apoint to point communications link, signal latency may be introducedwhich can result in an inconsistency in the sensed movement of theoutput compared with the input.

Furthermore, sensory devices require frequent updates in both signaldirections if the feel of the sensed movement and reaction to usersresponse is to be realistic. The number of updates required to maintainrealism, while not a problem where the haptic output device is in closeproximity and direct connection to the generating processor, may resultin the communications network overloading in a very short time and/ormay require extreme allocations of bandwidth.

According to the present invention there is provided a method ofactivating a haptic output device of the kind responsive to signalsdefining directional force comprising receiving a series of signalsdefining a multiplicity of data packets, each packet defining adirectional force applied at one location for transmission to thecurrent location, determining from packet data the information defininga position to which a haptic output device is expected to move, usingcurrent positional data to generate output signals defining force anddirection and applying a damping factor to said force and directionsignals to slow the rate of movement from a previously defined positiontowards the current defined position.

Preferably the method includes signalling in each direction wherebyhaptic forces applied at one device in reaction to an applied forcetowards the current defined position are reflected to a correspondingdevice in the form of current positional signals in a series of returndata packets.

A feature of the present invention provides an interactive haptic outputterminal in combination with a bidirectional transmission arrangement,the terminal comprising at least a haptic output device and controlmeans, said control means receiving signals from said haptic outputdevice to determine a current position for said device, and to determinefrom signals received from said transmission arrangement a preferredcurrent position for said haptic output device, said control meandetermining an output force and direction required to move said hapticoutput device from the current position to the preferred position andfurther modifying said output force and direction using a damping factorto slow apparent movement between the positions and outputting signalsdefining the modified force and direction.

The control means will receive signals from the haptic output devicecontaining data defining the position of said device at any particulartime and will convert said data to signals for transmission to saidbi-directional transmission arrangement at predetermined intervals.

The signals defining a preferred current position may be generated by anenvironment simulator, for example a programmed computer, or may begenerated by a corresponding interactive output terminal at the opposedend of the transmission arrangement.

Where a series of packets defining preferred position are received, eachpacket defining a directional force applied at one location fortransmission to the current location, the control means may includemeans to determine from packet data the sequence of transmission andre-ordering the data into a numerically correct series, extrapolatingfrom previously received packets an anticipated linear movement to bedefined by subsequently received packets and applying output directionalforce signals corresponding to said anticipated linear movement inrespect of any missing data packet.

A haptics communications system using the method of the invention andincorporating the apparatus according to the invention will now bedescribed by way of example only with reference to the accompanyingdrawings of which:

FIG. 1 is a block schematic diagram of a first haptics communicationssystem in which a network interconnects an environmental simulation to ahaptics input/output device;

FIG. 2 is a block schematic diagram of a haptics communications systemhaving a plurality of interconnected haptics input/output devices;

FIG. 3 is a schematic diagram of data interchange within the system ofFIG. 2;

FIG. 4 is a schematic flow chart of the method of measuring latencybetween two locations to effect adjustment of the system of FIG. 2;

FIG. 5 is a schematic flow chart of the method of calculating forces tobe applied locally;

FIG. 6 is a schematic flow chart of one way of adapting the calculationof FIG. 5; and

FIG. 7 shows a further adaptation of the algorithms of FIGS. 5 and 6.

Referring to FIG. 1, in our co-pending European patent applicationnumber 01305947.2, there is disclosed a method of providing a hapticsoutput representation of a scene stored, for example, as object modeldata. In this case a processor 1 includes a program responsive to theposition of a haptics output device (for example the Phantom 1.0 HapticOutput device from Sense Able Technologies Inc of the USA), to outputreaction the forces based upon the object model data. The object modeldata stored in a data store 3 could define textures, surfaces andlocations of fixed or moveable objects which could be perceived by auser of the haptics output interface 2. In some further developmentsdisclosed in the preceding application information held in a data store4 based upon a player identity 6 allowed player preferences 7 and agamma correction factor 8 to be used to provide appropriate outputadjustment to ensure that different players have approximately the sameperception of the output at the haptics output interface 2.

As disclosed the processor 1 was closely coupled to the haptics outputinterface 2 and could therefore provide substantially continuousdetection of location of the user's finger with respect to the x, y, zaxes of the device thus allowing real time simulation of the environmentdefined by the object model data 3.

As hereinbefore mentioned, once a network 5 is introduced between thehaptics output interface 2 and the processor 1 continuous communicationof the virtual environment and responsive signalling determining theuser's response and location by way of an input/output interface 6 tothe processor 1 becomes impractical if one requires to update thesignalling at substantially continuous rates. Furthermore, latencyintroduced to the signalling by the network results in an extremelyjerky feel to the information being transmitted.

Turning now to FIG. 2, where a plurality of haptic output devices 21, 22are communicating by way of respective input/output interfaces 25, 26 torespective processors 23, 24 the problems of network latency andsignalling limitations become more acute. Thus, if the processor 23receives by way of a network adaptor 27 signals indicating a positionfor the haptic output device 22 and instantly seeks to move the hapticdevice 21 to that position accordingly a substantial jerk in themovement will be apparent. In any event, the user of the haptic outputdevice 21 will be applying a backward force which may inhibit suchmovement and therefore prevent the processor 23 from aligning theposition of the haptic output device 21 with that of the haptic outputdevice 22. Correspondingly, the processor 23 in measuring the locationof the haptic output device 21 will send signals back through thenetwork 5 by way of network adaptor 28 to the processor 24, which willattempt to make a corresponding movement in the haptic output device 22.Thus, because the communication between haptic devices 21 and 22 is nolonger of a continuous mode but is receiving and transmitting positionalinformation at intervals the experience of the users will besignificantly impaired. In addition, the period of time taken forsignals to traverse the network (network latency) will further impairuser perception.

Thus considering also FIG. 3, it is possible to perceive that if at alocation A a user locates the haptic output device 21 “position” andreceives a force from the haptic output device then the local positiondata derived from “position” is derived by the personal computer (PC) 23and transmitted to the network. At the same time remote position datareceived from the network is translated by the PC 23 into local forcedata.

Corresponding position and force derivative data will also be used atlocation B by the PC 24.

In a practical network the position data and force output aretransmitted between each end at approximately five millisecondintervals. Thus each time a new position is received a force is outputin an attempt to move the output device to a new position, effectivelywith a motor pushing against the local user. In effect the user's ateach end at positions A and B are coupled together and the two-wayactivity and data transmission effectively attempts to move both outputdevices 21 and 22 to corresponding positions. This simulates connectionof the two users in a manner such as if they were connected together bysome kind of resilience device, for example, a spring or flexible rod.Because there is a reactionary force at each end there can beinstability between the two devices because of the effect of feed backloop so that deferred transmission of forces may result in an effectivepositive feedback.

In the present invention latency induced instability is damped down byeffectively introducing in the PCs 23 and 24 a simulated increase in theinertia of the respective connected haptic output device. Accordingly,turning to FIG. 4, from each of the locations A and B the respectivelocal clock 30 of the PCs 23,24 is used to determine the networklatency. Thus, from location A the time from the local clock is bundledinto a transmission packet, step 31, and transmitted at step 32 throughthe network 5. The packet is received at location B, step 33, and isimmediately retransmitted at step 34 through the network 5 and is againreceived at step 35 at location A the received time stripped out (thisbeing the time at which transmission first occurred) and the receivedtime is compared again at step 37 with the local clock 30 to provide, atstep 38, a usable measure of latency of the network 5.

Whilst it would be possible to transmit latency information across thenetwork so that each end used the same latency, in the case of networkvariations a similar latency measurement may be carried out fromlocation B as indicated using the respective local clock 40 to derive alatency measure by way of steps 41 to 47 corresponding to those of steps31 to 37.

It will be noted that because only one clock is involved in determiningthe latency measurement synchronisation of the clocks across the twocommunicating systems is not required. It will of course be appreciatedthat the packetisation need not necessarily be of specific clock timebut may simply be a serial number which is transmitted and received anda look up table is used to determine the time of transmission of theseries number packet for comparison with the current time.

Turning now to FIG. 5, once the latency in each direction has beendetermined, two methods of countering the latency problem may be used.In one instance a latency measurement is used to react negatively at thelocal end to modify the position data for transmission to the opposedend.

Considering FIG. 5, in a typical haptic coupling across the network,local positions derived from the haptic output device sensors asindicated at step 51 and the remote position received from the network,step 52, are used to calculate differences and to provide differencevector (step 53) in respect of the x, y and z coordinates of the twohaptic output devices. The coupling strength or resilience of thecoupling between the two devices is then used (step 54) to calculate theforce required to coerce the local haptic device to the relativeposition of the remote device (step 55) so that x, y and z vectors canbe transmitted to provide the local force for motors at step 56.

Referring now to FIG. 6, particularly in the invention , aftercalculation of the force required to coerce a local hand piece to theremote hand pieces relative position, the position history recorddetermining input movement is used to calculate acceleration, step 57,of the local haptic output device and the previously determined latencymeasure, step 58, is used to calculate an inertial value to be added inwhen determining the forces to be applied. In the alternative a look uptable may be used to provide a link between a measured latency value andan appropriate inertial value. The acceleration and inertial values cannow be used to provide an inertial force characteristic which can beused in combination with the calculated force for coercing the localhand piece to the remote hand piece relative position, step 61, so thatthe x, y and z forces output to the motors of the local haptics outputdevice are now adjusted to take account of latency or output.

Thus any tendency towards jerkiness is slightly smoothed by theintroduction of the inertial forces at step 60 and the probability ofthe establishment of a feed back loop substantially reduced.

Turning to FIG. 7, in a more complex arrangement it is possible tointroduce in addition to coupling strengths on latency measurements userpreferences. Thus, for example, in calculating the force required tocoerce a local hand piece to the relative position of a remote sensoruser preferences may affect the coupling strength 54, which can beadjusted (step 70) to take account of the position history and userpreferences.

The user preferences 72 may also be used to amend damping values 73 anddynamic variables 77 including but not limited to acceleration can becalculated as before with damping force 74 being used to take account oflocal hardware parameters 75 as well as user preferences and latencywith all such forces 76 contributing to the adaptation of the outputsignal data to local force motors 78.

The packets carrying the time information can be the same ones thatcarry the haptic effector position information. This has the advantageof ensuring the appropriate network stack delays for the positioninformation are included in the measured latency and reduces the numberof packets which need to be sent.

There are other physical parameters besides mass which could be adjustedto damp latency induced instability. These include friction viscous drag(force proportional to & opposing velocity) & friction (a force opposingvelocity and of constant magnitude if velocity non-zero, otherwise ofequal magnitude to & opposing the applied force up to a limit).

Instead of using the parameters related to the physics of the hapticeffector, those related to the coupling between the haptic effectorscould be used. For example, decreasing the strength with which eacheffector tries to jump to the position of the other can reduceinstability.

The different damping means mentioned above could be applied incombination as well as individually.

Other methods of haptic transmission (e.g. measuring reaction force fromthe environment at one effector, transmitting it and outputting it viathe other effector) could be used instead of coercing the effectorhandpieces to each-other's position using a simulated spring force.

Other methods of networking latency measurement (e.g. ISDN, TCP over IPor RS232 serial over a modem to modem link over PSTN) could be usedinstead of UDP.

Other methods of network latency measurement (e.g. ‘ping’ time, networkperformance metrics from other computers on the network, or singledirection measurement by synchronised clocks) could be used.

Other forms of adjustable instability suppression could be adjusted inresponse to latency measurements.

The simulated mass (or other damping parameter) could be adjustedautomatically by means other than latency measurement. For example itcould be automatically increased whenever the onset of oscillationsindicating instability are detected and gradually decreased when noinstability has been detected for a period of time. The forms ofinstability damping described could be used without the automaticadjustment, e.g. the simulated mass kept at a value which predefined orset by the user.

Instability is also affected by how a user responds to the effectorsmotion. In our trials, some users were able to learn to dampen someinstability themselves. Such users might therefore prefer less dampingso as to increase speed & detail of feeling at the expense of stability.Therefore it is proposed to include in the calculation of the degree ofdamping a parameter which the user has some control over.

Alternatively the system could adapt the degree of damping to fit theuser automatically (e.g. by decreasing the damping until the start ofoscillations indicating instability are detected).

The per-user parameters, whether manually or automatically set, could bestored for future use.

The adjustments made to the position parameters can be combined withother senses. Most obviously is a visual representation of a hapticscene but also with sound, smell and other perceptive stimuli.

1. A method of activating a haptic output device of the kind responsiveto signals defining directional force comprising receiving a series ofsignals defining a multiplicity of data packets, each packet defining adirectional force applied at one location for transmission to thecurrent location, determining from packet data the information defininga position to which a haptic output device is expected to move, usingcurrent positional data to generate output signals defining force anddirection and applying a damping factor to said force and directionsignals to slow the rate of movement from a previously defined positiontowards the current defined position.
 2. The method of claim 1 furtherincluding signalling in each direction whereby haptic forces applied atone device in reaction to an applied force towards the current definedposition are reflected to a corresponding device in the form of currentpositional signals in a series of return data packets.
 3. The method ofclaim 1 further including calculating the damping factor from determinedparameters of a transmission network on which said data packets arecarried.
 4. The method of claim 3 in which latency of the network isdetermined by transmitting a data packet to the network said packetincluding a time determinant identity, reflecting the data packetthrough the network and comparing the received time with the transmittedtime to provide a latency parameter from which said damping factor isdetermined.
 5. The method of claim 4 in which at least some transmittedpackets carrying positional data also include the time determinant data,some of said time determinant data being returned to permit updating ofthe latency parameter.
 6. The method of claim 1 further includingapplying a weighting factor in addition to the damping factor, theweighting factor being derived from other parameters of theinterconnection such as resilience.
 7. The method of claim 1 furtherincluding applying a modifying factor to the force and directionsignals, said modifying factor being derived from pre determined userpreference data.
 8. An interactive haptic output terminal in combinationwith a bi-directional transmission arrangement, the terminal comprisingat least a haptic output device and control means, said control meansreceiving signals from said haptic output device to determine a currentposition for said device, and to determine from signals received fromsaid transmission arrangement a preferred current position for saidhaptic output device, said control mean determining an output force anddirection required to move said haptic output device from the currentposition to the preferred position and further modifying said outputforce and direction using a damping factor to slow apparent movementbetween the positions and outputting signals defining the modified forceand direction.
 9. A terminal as claimed in claim 8 in which the controlmeans receives signals from the haptic output device, said signalscontaining data defining the position of said device at any particulartime, said control means converting said data to signals fortransmission to said bi-directional transmission arrangement atpredetermined intervals.
 10. A terminal as claimed in claim 8 in whichthe signals defining a preferred current position are generated by anenvironment simulator, for example a programmed computer.
 11. A terminalas claimed in claim 8 in which the signals defining a preferred currentposition are generated by a corresponding interactive output terminal atthe opposed end of the transmission arrangement.
 12. A terminal asclaimed in claim 8 in which a series of packets defining preferredposition are received, each packet defining a directional force appliedat one location for transmission to the current location, the controlmeans includes means to determine from packet data the sequence oftransmission and re-ordering the data into a numerically correct series,extrapolating from previously received packets an anticipated linearmovement to be defined by subsequently received packets and applyingoutput directional force signals corresponding to said anticipatedlinear movement in respect of any missing data packet.
 13. A method ofactivating a haptic output device as claimed in claim 1 includingsignalling in each direction whereby haptic forces applied at one devicein reaction to an applied force towards the current defined position arereflected to a corresponding device in the form of current positionalsignals in a series of return data packets.
 14. An interactive hapticoutput terminal in combination with a bi-directional transmissionarrangement, the terminal comprising at least a haptic output device andcontrol means, said control means receiving signals from said hapticoutput device to determine a current position for said device, and todetermine from signals received from said transmission arrangement apreferred current position for said haptic output device, said controlmean determining an output force and direction required to move saidhaptic output device from the current position to the preferred positionand further modifying said output force and direction by applying adamping factor to slow apparent movement between the positions andoutputting signals defining the modified force and direction.
 15. Aterminal as claimed in claim 14 in which the control means receivessignals from the haptic output device containing data defining theposition of said device at any particular time and converts said data tosignals for transmission to said bi-directional transmission arrangementat pre-etermined intervals.
 16. A terminal as claimed in claim 14 inwhich the signals defining a preferred current position are generated byan environment simulator, for example a programmed computer.
 17. Aterminal as claimed in claim 14 in which the signals defining apreferred current position are generated by a corresponding interactiveoutput terminal at the opposed end of the transmission arrangement. 18.A terminal as claimed in claim 14 in which a series of packets definingpreferred position are received, each packet defining a directionalforce applied at one location for transmission to the current location,the control means including means to determine from packet data thesequence of transmission and re-ordering the data into a numericallycorrect series, extrapolating from previously received packets ananticipated linear movement to be defined by subsequently receivedpackets and applying output directional force signals corresponding tosaid anticipated linear movement in respect of any missing data packet.